Foundation models are redefining how AI systems are built. Practitioners now follow a standard procedure to build their machine learning solutions: download a copy of a foundation model, and fine-tune it using some in-house data about the target task of interest. Consequently, the Internet is swarmed by a handful of foundation models fine-tuned on many diverse tasks. Yet, these individual fine-tunings often lack strong generalization and exist in isolation without benefiting from each other. In our opinion, this is a missed opportunity, as these specialized models contain diverse features. Based on this insight, we propose model recycling, a simple strategy that leverages multiple fine-tunings of the same foundation model on diverse auxiliary tasks, and repurposes them as rich and diverse initializations for the target task. Specifically, model recycling fine-tunes in parallel each specialized model on the target task, and then averages the weights of all target fine-tunings into a final model. Empirically, we show that model recycling maximizes model diversity by benefiting from diverse auxiliary tasks, and achieves a new state of the art on the reference DomainBed benchmark for out-of-distribution generalization. Looking forward, model recycling is a contribution to the emerging paradigm of updatable machine learning where, akin to open-source software development, the community collaborates to incrementally and reliably update machine learning models.

随着学习机器对人类生命的决策的影响，分析其公平性能成为重要性的主题。然而，我们衡量学习系统公平性的最佳工具是将数学单线封装的僵化公平指标，为参与预测任务的利益相关者提供有限的权力，并且当我们敦促过度压力以优化它们时，很容易操纵。为了促进这些问题，我们建议将重点从塑造公平指标转变为策划计算这些示例的分布。特别是，我们认为，关于公平性的每一个主张都应立即遵循标语“在哪些例子下公平，并由谁收集？”。通过强调与域概括中文献的联系，我们建议衡量公平性作为系统在多个压力测试下概括的能力 - 具有社会相关性的示例的分布。我们鼓励每个利益相关者策划一个或多个压力测试，其中包含反映其（可能相互矛盾的）利益的例子。该机器通过缺乏或超过预定义的度量值而通过或未能使每个压力测试失败。测试结果涉及所有利益相关者参与有关如何改善学习系统的讨论，并根据上下文和基于可解释的数据提供对公平性的灵活评估。我们为压力测试提供了完整的实施指南，既说明了该框架的好处和缺点，又引入了一个加密计划，以使系统提供商获得一定程度的预测问责制。

在易于优化和强大的分布（OOD）概括之间通常存在困境。例如，许多OOD方法依赖于优化具有挑战性的罚款术语。他们要么太强大，无法可靠地优化，要么太虚弱而无法实现目标。我们建议用丰富的表示，其中包含一个潜在有用功能的调色板初始化网络，即使是简单的模型也可以使用。一方面，丰富的表示为优化器提供了良好的初始化。另一方面，它还提供了有助于OOD概括的电感偏差。这种表示形式是由丰富的功能构建（RFC）算法（也称为盆景算法）构建的，该算法由一系列培训情节组成。在发现剧集中，我们以防止网络使用以前迭代中构建的功能的方式制作了多目标优化标准及其相关数据集。在合成事件中，我们使用知识蒸馏来迫使网络同时代表所有先前发现的特征。用盆景表示的网络初始化，始终有助于六种OOD方法在ColoredMnist基准上实现最佳性能。相同的技术在Wilds Camelyon17任务上大大优于可比较的结果，消除了困扰其他方法的高结果差异，并使超参数调谐和模型选择更加可靠。

The goal of domain generalization algorithms is to predict well on distributions different from those seen during training. While a myriad of domain generalization algorithms exist, inconsistencies in experimental conditions-datasets, architectures, and model selection criteria-render fair and realistic comparisons difficult. In this paper, we are interested in understanding how useful domain generalization algorithms are in realistic settings. As a first step, we realize that model selection is non-trivial for domain generalization tasks. Contrary to prior work, we argue that domain generalization algorithms without a model selection strategy should be regarded as incomplete. Next, we implement DOMAINBED, a testbed for domain generalization including seven multi-domain datasets, nine baseline algorithms, and three model selection criteria. We conduct extensive experiments using DO-MAINBED and find that, when carefully implemented, empirical risk minimization shows state-of-the-art performance across all datasets. Looking forward, we hope that the release of DOMAINBED, along with contributions from fellow researchers, will streamline reproducible and rigorous research in domain generalization. * Alphabetical order, equal contribution.Preprint. Under review.

Deep neural networks excel at learning the training data, but often provide incorrect and confident predictions when evaluated on slightly different test examples. This includes distribution shifts, outliers, and adversarial examples. To address these issues, we propose Manifold Mixup, a simple regularizer that encourages neural networks to predict less confidently on interpolations of hidden representations. Manifold Mixup leverages semantic interpolations as additional training signal, obtaining neural networks with smoother decision boundaries at multiple levels of representation. As a result, neural networks trained with Manifold Mixup learn class-representations with fewer directions of variance. We prove theory on why this flattening happens under ideal conditions, validate it on practical situations, and connect it to previous works on information theory and generalization. In spite of incurring no significant computation and being implemented in a few lines of code, Manifold Mixup improves strong baselines in supervised learning, robustness to single-step adversarial attacks, and test log-likelihood.

本文提出了一种新的因果发现方法，即结构不可知的建模（SAM）。SAM利用条件独立性和分布不对称性，旨在从观察数据中找到潜在的因果结构。该方法基于不同玩家之间的游戏，该游戏将每个变量分布有条件地作为神经网估算，而对手则旨在区分生成的数据与原始数据。结合分布估计，稀疏性和无环限制的学习标准用于通过随机梯度下降来实施图形结构和参数的优化。SAM在合成和真实数据上进行了实验验证。

Large deep neural networks are powerful, but exhibit undesirable behaviors such as memorization and sensitivity to adversarial examples. In this work, we propose mixup, a simple learning principle to alleviate these issues. In essence, mixup trains a neural network on convex combinations of pairs of examples and their labels. By doing so, mixup regularizes the neural network to favor simple linear behavior in-between training examples. Our experiments on the ImageNet-2012, CIFAR-10, CIFAR-100, Google commands and UCI datasets show that mixup improves the generalization of state-of-the-art neural network architectures. We also find that mixup reduces the memorization of corrupt labels, increases the robustness to adversarial examples, and stabilizes the training of generative adversarial networks.

One major obstacle towards AI is the poor ability of models to solve new problems quicker, and without forgetting previously acquired knowledge. To better understand this issue, we study the problem of continual learning, where the model observes, once and one by one, examples concerning a sequence of tasks. First, we propose a set of metrics to evaluate models learning over a continuum of data. These metrics characterize models not only by their test accuracy, but also in terms of their ability to transfer knowledge across tasks. Second, we propose a model for continual learning, called Gradient Episodic Memory (GEM) that alleviates forgetting, while allowing beneficial transfer of knowledge to previous tasks. Our experiments on variants of the MNIST and CIFAR-100 datasets demonstrate the strong performance of GEM when compared to the state-of-the-art.

In this paper, we propose a novel technique, namely INVALIDATOR, to automatically assess the correctness of APR-generated patches via semantic and syntactic reasoning. INVALIDATOR reasons about program semantic via program invariants while it also captures program syntax via language semantic learned from large code corpus using the pre-trained language model. Given a buggy program and the developer-patched program, INVALIDATOR infers likely invariants on both programs. Then, INVALIDATOR determines that a APR-generated patch overfits if: (1) it violates correct specifications or (2) maintains errors behaviors of the original buggy program. In case our approach fails to determine an overfitting patch based on invariants, INVALIDATOR utilizes a trained model from labeled patches to assess patch correctness based on program syntax. The benefit of INVALIDATOR is three-fold. First, INVALIDATOR is able to leverage both semantic and syntactic reasoning to enhance its discriminant capability. Second, INVALIDATOR does not require new test cases to be generated but instead only relies on the current test suite and uses invariant inference to generalize the behaviors of a program. Third, INVALIDATOR is fully automated. We have conducted our experiments on a dataset of 885 patches generated on real-world programs in Defects4J. Experiment results show that INVALIDATOR correctly classified 79% overfitting patches, accounting for 23% more overfitting patches being detected by the best baseline. INVALIDATOR also substantially outperforms the best baselines by 14% and 19% in terms of Accuracy and F-Measure, respectively.

The recent increase in public and academic interest in preserving biodiversity has led to the growth of the field of conservation technology. This field involves designing and constructing tools that utilize technology to aid in the conservation of wildlife. In this article, we will use case studies to demonstrate the importance of designing conservation tools with human-wildlife interaction in mind and provide a framework for creating successful tools. These case studies include a range of complexities, from simple cat collars to machine learning and game theory methodologies. Our goal is to introduce and inform current and future researchers in the field of conservation technology and provide references for educating the next generation of conservation technologists. Conservation technology not only has the potential to benefit biodiversity but also has broader impacts on fields such as sustainability and environmental protection. By using innovative technologies to address conservation challenges, we can find more effective and efficient solutions to protect and preserve our planet's resources.